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Interactive Virtual Endoscopy 



Field of Invention 

The invention relates to a method for processing a three-dimensional image data set, 
wherein the three-dimensional image data set is converted to a data set for two- 
dimensional image reproduction. The method is particularly appropriate for virtual en- 
doscopy (e.g., coloscopy). The invention further relates to apparatuses for performing 
the required calculations and/or the reproduction of the data representations. 

Background Art 

In medical science, the endoscopy of a colon by means of an endoscope is refen^ed to 
as coloscopy and the respective specifically designed endoscopes are known as colo- 
scopes. A coloscope comprises an optical system usually connected with a display de- 
vice to allow for diagnosis by a physician. Insertion of a coloscope in the colon is sensed 
as being intrusive or even painful by many patients and always, in particular, in the case 
of enteritis/colitis, includes a risk of penetrating the intestinal wall. 

As an alternative virtual coloscopy has been developed during which no physical colo- 
scope has to be inserted in the patient's body any longer. Instead of coloscopes meth- 
ods and apparatuses used in computer tomography / magnetic resonance imaging are 
used to detect data and to visualize the data. Progress of virtual coloscopy has signifi- 
cantly been advanced by the fact that due to the increased computational power of pre- 
sent day computers expensive image processing can readily be carried out. 

With virtual coloscopy, a large number of spatially resolved parallel slices is recorded by 
a tomographic means. Each of these slices corresponds to a set of two-dimensional im- 
age data. These data sets are numerically converted to a three-dimensional image data 
set. From this three-dimensional image, data set two-dimensional image data can be 
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calculated again that are independent of the direction of the original slice (cut), e.g., 
oblique to this direction. Usually the two- and three-dimensional image data are repro- 
duced by means of a two-dimensional display device (monitor, photo, etc.), namely they 
are visualized as sectional views (i.e. all of the displayed pixels belong to one intersec- 
tion plane) or as pseudo-three-dimensional images that create a three-dimensional im- 
pression in a way similar to conventional photography (the pixels do not originate from a 
plane surface). 

It should be noted that it could not be decided a priori which one of the views is most 
suitable for the most accurate diagnosis. Whereas the (pseudo)-three-dimensional data 
representations are rather demonstrative due to the spatial impression given and thus, 
are helpful for the spatial orientation, it is particulariy for these views that possible 
anomalies as lesions are hidden by tissue (e.g. by a colon fold) and therefore are invisi- 
ble. Contrarily, in sectional views such a concealing does not occur. However, the sec- 
tional views are not very helpful for a medical treatment that usually requires a three- 
dimensional coordination. 

Description of the Invention 

In view of the above problems it is an object of the invention to provide methods and 
apparatuses that do not suffer from the mentioned drawbacks of virtual coloscopy. 

This object is solved by the method according to claim 1 , the computer program product 
according to daim 9 and the image processing and reproduction system having the 
features of claim 10. Advantageous embodiments are defined in the dependent claims. 

Whereas the invention is made specifically for the needs of virtual coloscopy and 
whereas it is described with regard to this range of application, it is to be understood that 
the basic ideas of the invention can be applied to any three-dimensional data set. Po- 
tential further ranges of application in medical science include all kinds of virtual endos- 
copy, other tomographic methods, ultrasonic examination methods, X-ray examination 
methods with tracers, etc. Potential ranges of application outside medical science com- 
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prise atmospheric research (e.g., meteorology, climatology), seismology, exploration of 
natural resources (ore, coal, oil and gas resources), material testing, etc. 

According to the inventive method a three-dimensional image data set is converted to at 
least two different data representations for image reproduction and a reproduction of the 
at least two different data representations is realized, wherein one image section that is 
shown in one of the data representations is mari<able and for the at least one other data 
representation the relative position of the merited image section is calculated and 
shown. 

Thus, image data for (at least) one sectional view, which is more suitable for detection of 
anomalies, and image data for (at least) one three-dimensional view, which is more suit- 
able for three-dimensional imagination, can be extracted from a three-dimensional data 
set obtained by virtual coloscopy, for example. Since a priori it can hardly be foreseen 
which one of the views is most indicative for diagnosis, employment of different views, in 
particular, multiple different sectional views and multiple different three-dimensional 
views, allow for a more reliable diagnosis. 

If one of the view shows an anomaly the anomalous section in this data representation 
can be marked, e.g., by means of a mouse click, by means of a maricing polygon or 
other marking methods. After the marking has been done it is automatically calculated 
where the mari<ed section is located in the other data representation or where it is lo- 
cated relatively to the other data representation respectively. 

If the section mariced in the first data representation is also visually shown in the second 
data representation, this can be indicated, e.g., by a solid contour, a solid circle etc. 
around the respective image section in the second data representation. If on the other 
hand the section is hidden by tissue in the second data representation or if the section is 
located above or below the shown surface in case of sectional views, this may be indi- 
cated by, e.g., a dashed maricing. This has the additional advantage that different physi- 
cians, e.g. physicians with different experiences, in particular, physicians from different 
fields (e.g. radiologists and gastroenterologists) whose standard wori<ing views are often 
different from each other can communicate more precisely with each other, since the 
risk of ambiguous diagnoses or interpretations is reduced. As a result this leads to an 
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improved collaboration of the physicians and to an improved as well as less costly medi- 
cal treatment. 

If the marking exhibits a specified geometric form, it is advantageous that this fomi is 
defomied for further data representations corresponding to changed line of sights so that 
the viewer gains a perspective view. 

The explanation given above for one further or one second data representation can be 
generalized for a plurality of further data representations. The different data representa- 
tions may, e.g., be simultaneously reproduced with one single reproduction device (e.g., 
a monitor) with all the data representations being displayed with the same dimensions or 
with different sizes according to their significance. 

The inventive method may be perfomned by exclusively employing either sectional views 
or three-dimensional views. 

The advantages of the inventive method, however, are particularly emphasized, if at 
least one data representation is a sectional view and at least one data representation is 
a three-dimensional view, since these data representations are somewhat complemen- 
tary to each other in teniis of advantages and disadvantages. 

Preferably, the at least one sectional view comprises an axial view and/or a frontal view 
and/or a sagittal view and/or an oblique view and the at least one three-dimensional 
view comprises a wall (mucous membrane) view and/or an intraluminal view. These 
views represent specific views commonly known by the skilled person that are described 
In more detail with reference to the accompanying figure. 

According to a pretended embodiment of the inventive method the at least two different 
data representations comprise a topogram view (as described in more detail with refer- 
ence to the accompanying figure) which offers an excellent overview of the section un- 
der examination and for which numerical methods for automatic visualization of the co- 
lon extension are available. 
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Thereby a position can be marl^ed in the shown colon extension at which an anomaly is 
suspected or that is to be examined for different reasons. The respective marked section 
or its relative position can then be shown in the other data representation(s). 

On the other hand, an anomaly found and marked in one other data representation may 
be indicated in the colon extension reproduced by the topogram and helpful infonnation 
with respect to diagnosis/therapy are thereby provided. 

If in one of the data representations an anomaly is found and marked, according to an 
advantageous embodiment, the other data representations can be changed in such a 
way that the image section shov^ in the one data representation is also shown in the 
other data representations. For example, the intersection planes for the (other) sectional 
views may be automatically selected such that they also visually show the marked 
anomaly, e.g., in such a manner that the anomaly occupies a maximum sectional area. 

According to a prefen^ed embodiment, the marking is done manually, e.g., by an input 
device as a computer mouse, a computer keyboard, a touchpad, etc. Alternatively or in 
addition, the marking of the shown image section can be done by rneans of automatic 
structure and/or texture recognizing methods that can identify anomalies in an image by 
particular characteristics of these anomalies. 

An embodiment of the method according to which the three-dimensional image data set 
of a hose-shaped body is processed and at least one of the data representations is a 
wall view with a line of sight that is parallel and/or anti-parallel to the curvature vector at 
the maximum curvature of the mid-line of the hose-shaped body is advantageous for the 
examination of hose-shaped bodies as canals and other leads, blood vessels and, in 
particular, for endoscopy. 

By calculating individual pixels that define a mid-line by calculating barycenters, such a 
mid-line can usually be defined for a hose-shaped body, even if the cross sections of the 
body differ from ideal circles. 

The maximum curvature may be understood as the absolute maximum in the mathe- 
matical sense but usually it is taken as a local maximum in the mathematical sense (lo- 
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cal maximum curvature) representing a position at which the curvature vector is smaller 
than in its closest proximity. 

The above-described method is particularly helpful in examining hose-shaped bodies 
that do not extend straight along one spatial direction but rather are (possibly repeat- 
edly) bent. The mid-line of such a body represents a curved line in space. 

ft 

In the case of bent hose-shaped bodies, the method provides at least one of the follow- 
ing wall views: the wall view of the area of the inner surface of the hose-shaped body 
that is bent most in the longitudinal direction and/or the opposite wall view. For bent ca- 
nals the shown wall views are, e.g., elongated or shortened walls sections at which the 
material is stressed most and that are therefore particularly vulnerable to defects. In the 
context of the colon or blood vessels such areas showing strong curvatures and even 
kinks are sections in which anomalies not only preferably occur but also are particularly 
difficult to detect. 

The above described inventive method and the preferred embodiments are particularly 
suitable for virtual endoscopy, in particular, virtual coloscopy (but also in neighbored 
fields as virtual bonchioscopy, intestinoscopy, NHN-endoscopy, ventricle endoscopy). 
By the interactive interplay of the different data representations one can profit from com- 
plementary advantages of the different data representations and disadvantages that are 
specific for each of the data representations can be circumvented. 

For example, the inventive method can also be applied to conventional coloscopy, since 
an ultrasound head at a conventional coloscope can also provide three-dimensional im- 
age data sets that are similar to the ones discussed above. 

The object underlying the invention is further solved by computer program product to be 
read by or to be implemented in a computer respectively and that is adapted to perfonn 
the steps of (a) converting a three-dimensional image data set to at least two different 
data representations for image reproduction and (d) calculating the relative position of 
the marked image section in the at least one other data representation. 
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The computer program product converts the three-dimensional image data set to differ- 
ent data representations for a reproduction of the images derived from the three- 
dimensional image data set. A marking applied ex post to one of the data representa- 
tions is processed so that a relative position of the marking in the other data representa- 
tions is calculated and that the respective image data are modified in accordance with 
the calculated values so that the relative position of the marking can be shown in the 
other data representations. 

The object is also solved by an image processing and reproducing system comprising at 
least one device, e.g., a monitor, for Image reproduction of a three-dimensional image 
data set by at least two different data representations, a device, e.g., a computer mouse, 
for marking one image section that is shown in one of the data representations; a device 
for calculation of the relative position of the image section marked in the one data repre- 
sentation in the at least one other data representation. 

Preferably, the above described computer program product is implemented in the device 
for calculation of the relative position of the image section marked in ttie one data repre- 
sentation in the at least one other data representation. 

In the following, the invention and the advantages of the invention will be described with 
respect to a preferred example and witii reference to the accompanying figure. 

Fig. 1 shows most beneficial views of ttie virtual coloscopy altogether integrated in one 
user interface. 

The upper row in Fig. 1 shows four two-dimensional views, namely from left to right: 

an axial view 1 conresponding to a cut perpendicular to the longitudinal axis of the 
body 

a frontal view 2, 
a sagittal view 3, and 
an oblique view 4. 
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The intersection planes of the three sectional views 1 , 2 and 3 are orientated perpen- 
dicular to each other. Orientation of the oblique view 4 is arbitrary. It is useful to orientate 
the surface nornial of the oblique view corresponding to an axis or the prefen^ed orienta- 
tion of the organ under examination. 

The two-dimensional sectional view 4 is calculated from the generated three- 
dimensional data set that again was generated from a plurality of two-dimensional sets 
of detected data. 

The data representations 5 to 8 do not represent sectional views but rather give a three- 
dimensional impression and thereby allow for an overview of the examined body region. 

The data representations 4 and 5 are called wall views. The images result from virtually 
cutting the colon canal parallel to the longitudinal axis and by taking pictures by virtual 
cameras that are orientated perpendicular to the longitudinal axis. One can distinguish 
between an "anterior wall view" 5 and a "posterior wall view" 6 for an opposite and a 
back view of the intestinal wall, respectively. 

Internists are familiar with views like 7 and 8 of Fig. 1 . They allow the experienced in- 
temenst to gain a three-dimensional impression of the region that is to be examined or to 
be treated. 

The intraluminal view 7 conresponds to an image of the inner colon canal obtained by 
conventional coloscopy with the difference that it is generated by means of a virtual co- 
loscope. 

The topogram view 8 that is also generated from a three-dimensional data set by the 
tomographic method corresponds to a common radiogram. The sinuous line seen in the 
figure represents a virtual path within the colon that is generated automatically by a 
computer. 

If. e.g., the physician recognizes an anomaly of the intestinal mucosa in one of the im- 
ages, he Is enabled to select the respective image section in the image by a mouse click 
and he subsequently is provided with the respective positions in all the other images, i.e. 
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the sectional views (two-dimensional images) as well as the (quasi-)three<limensional 
data representations. 

In the (quasi-)three-dimensional data representations a marking appears, if the anomaly, 
e.g., a lesion, is visible for the virtual camera, and a dashed marking appears, if it is not 
visible for the virtual camera. Thus, the internist, who possibly has to canry out the treat- 
ment, learns behind which tissue (e.g., a colon fold) the anomaly is hidden. The views 
"intraluminar and "topogram" are of particular importance, since they represent the kind 
of data representations internists are usually familiar with and they are by far most ap- 
propriate for three-dimensional orientation in terms of lucidity and clearness. Supple- 
mented by a marking in the topogram they allow for an immediate localization of an 
anomaly by the intemist. 
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